Greptile Summary

This PR introduces JobLauncherSpec / JobHandleSpec abstraction interfaces, a Kubernetes-backed implementation (K8sJobLauncher / K8sJobHandle), two new Best-Effort resource management classes (BEResourceManager, BEResourceConsumer), an ignore_host flag for GPUResourceManager to support K8s deployments, a comprehensive unit-test suite, and a design document.

Key changes:

• nvflare/apis/job_launcher_spec.py: Clean new spec layer defining JobHandleSpec, JobLauncherSpec, JobProcessArgs, and JobReturnCode. Well-structured and consistent with existing NVFlare component conventions.
• nvflare/app_opt/job_launcher/k8s_launcher.py: Full K8s launcher. Several bugs flagged in earlier review rounds have been fixed in this version — None module-arg values are now skipped when building pod container args, terminate() sets terminal_state in all code paths, wait() persists the terminal state so subsequent poll() calls remain accurate, and the return value of enter_states() is now checked in launch_job(). One remaining suggestion: emit a logger.warning when a data-PVC YAML file contains more than one entry, since currently only the first key is used and extras are silently discarded.
• Resource managers: BEResourceManager and BEResourceConsumer are minimal, correct best-effort implementations. The new ignore_host parameter in GPUResourceManager cleanly addresses the split-plane K8s topology.
• Tests: Coverage is thorough across UUID conversion, pod-state/return-code mapping, full manifest construction, stuck-detection, wait/poll/terminate lifecycle, and launcher initialization. Two minor test-quality issues in configer_test.py: an assertion inside the for-loop in test_merge_configs exits on the first mismatch rather than reporting all failures, and test_add_and_remove_config_keys uses list(file_merged.items())[0] which assumes dict insertion order for its correctness.

Confidence Score: 4/5

• This PR is safe to merge with minor style improvements recommended.
• The implementation is well-structured and the majority of bugs identified in prior review rounds have been addressed in the current HEAD. The remaining comments are style-level (silent PVC key selection, hardcoded poll interval, fragile test assertions) rather than correctness bugs. The unit test coverage is comprehensive and the design doc matches the implementation.
• nvflare/app_opt/job_launcher/k8s_launcher.py (data PVC silent first-key selection), tests/unit_test/tool/job/config/configer_test.py (fragile test assertions)

Important Files Changed

Sequence Diagram

sequenceDiagram
    participant UL as Upper Layer<br/>(ServerEngine/ClientExecutor)
    participant EH as Event System
    participant KL as K8sJobLauncher
    participant KH as K8sJobHandle
    participant K8s as Kubernetes API

    UL->>EH: fire BEFORE_JOB_LAUNCH event
    EH->>KL: handle_event(BEFORE_JOB_LAUNCH, fl_ctx)
    KL->>KL: extract_job_image(job_meta, site_name)
    alt job_image present
        KL->>EH: add_launcher(self, fl_ctx)
    end

    UL->>KL: launch_job(job_meta, fl_ctx)
    KL->>KL: uuid4_to_rfc1123(raw_job_id)
    KL->>KL: get_module_args(job_id, fl_ctx)
    KL->>KH: K8sJobHandle(job_id, core_v1, job_config)
    KH->>KH: _make_manifest(job_config)
    KH-->>KL: job_handle

    KL->>K8s: create_namespaced_pod(pod_manifest)
    alt pod creation fails
        K8s-->>KL: Exception
        KL->>KH: terminal_state = TERMINATED
        KL-->>UL: job_handle (terminated)
    else pod created
        K8s-->>KL: ok
        KL->>KH: enter_states([RUNNING])
        loop poll until RUNNING or timeout
            KH->>K8s: read_namespaced_pod(job_id)
            K8s-->>KH: pod phase
            alt stuck in PENDING
                KH->>K8s: delete_namespaced_pod(job_id)
                KH->>KH: terminal_state = TERMINATED
            end
        end
        KH-->>KL: True/False
        KL-->>UL: job_handle
    end

    UL->>KH: wait()
    loop until terminal
        KH->>K8s: read_namespaced_pod(job_id)
        K8s-->>KH: phase (SUCCEEDED/FAILED/RUNNING)
        alt SUCCEEDED or FAILED
            KH->>KH: terminal_state = job_state
        end
    end

    UL->>KH: poll()
    KH-->>UL: JobReturnCode (SUCCESS/ABORTED/UNKNOWN)

_{Last reviewed commit: "Fix reference error"}

/build

/build